• 41 Posts
  • 264 Comments
Joined 2 years ago
Cake day: June 9th, 2023
  • j4k3@lemmy.worldtoSelfhosted@lemmy.world•Consumer GPUs to run LLMsEnglish
    5·
    4 hours ago
    Anything under 16 is a no go. Your number of CPU cores are important. Use Oobabooga Textgen for an advanced llama.cpp setup that splits between the CPU and GPU. You'll need at least 64 GB of RAM or be willing to offload layers using the NVME with deepspeed. I can run up to a 72b model with 4 bit quantization in GGUF with a 12700 laptop with a mobile 3080Ti which has 16GB of VRAM (mobile is like that).

    I prefer to run a 8×7b mixture of experts model because only 2 of the 8 are ever running at the same time. I am running that in 4 bit quantized GGUF and it takes 56 GB total to load. Once loaded it is about like a 13b model for speed but is ~90% of the capabilities of a 70b. The streaming speed is faster than my fastest reading pace.

    A 70b model streams at my slowest tenable reading pace.

    Both of these options are exponentially more capable than any of the smaller model sizes even if you screw around with training. Unfortunately, this streaming speed is still pretty slow for most advanced agentic stuff. Maybe if I had 24 to 48gb it would be different, I cannot say. If I was building now, I would be looking at what hardware options have the largest L1 cache, the most cores that include the most advanced AVX instructions. Generally, anything with efficiency cores are removing AVX and because the CPU schedulers in kernels are usually unable to handle this asymmetry consumer junk has poor AVX support. It is quite likely that all the problems Intel has had in recent years has been due to how they tried to block consumer stuff from accessing the advanced P-core instructions that were only blocked in microcode. It requires disabling the e-cores or setting up a CPU set isolation in Linux or BSD distros.

    You need good Linux support even if you run windows. Most good and advanced stuff with AI will be done with WSL if you haven’t ditched doz for whatever reason. Use https://linux-hardware.org/ to see support for devices.

    The reason I mentioned avoid consumer e-cores is because there have been some articles popping up lately about all p-core hardware.

    The main constraint for the CPU is the L2 to L1 cache bus width. Researching this deeply may be beneficial.

    Splitting the load between multiple GPUs may be an option too. As of a year ago, the cheapest option for a 16 GB GPU in a machine was a second hand 12th gen Intel laptop with a 3080Ti by a considerable margin when all of it is added up. It is noisy, gets hot, and I hate it many times, wishing I had gotten a server like setup for AI, but I have something and that is what matters.

  • j4k3@lemmy.worldtoAsklemmy@lemmy.ml•Have you ever had someone get mad at you for asking why you shouldn't do something?English
    1·
    4 hours ago
    I like to write, but have never done so professionally. I disagree that it hurts writers. I think people reacted poorly to AI because of the direct and indirect information campaign Altmann funded to try and make himself a monopoly. AI is just a tool. It is fun to play with in unique areas, but these often require very large models and/or advanced frameworks. In my science fiction universe I must go to extreme lengths to get the model to play along with several aspects like a restructure of politics, economics, and social hierarchy. I use several predictions I imagine about the distant future that plausibly make the present world seem primitive in several ways and with good reasons. This restructuring of society violates both some of our cultural norms in the present and is deep within areas of politics that are blocked by alignment. I tell a story where humans are the potentially volatile monsters to be feared. That is not the plot, but convincing a present model to collaborate on such a story ends up in the gutter a lot. My grammar and thought stream is not great and that is the main thing I use a model to clean up, but it is still collaborative to some extent.

    I feel like there is an enormous range of stories to tell and that AI only makes these more accessible. I have gone off on tangents many times exploring parts of my universe because of directions the LLM took. Like I limit the model to generate a sentence at a time and I’m writing half or more of every sentence for the first 10k tokens. Then it picks up on my style so much that I can start the sentence with a word or change one word in a sentence and let it continue with great effect. It is most entertaining to me because it is almost as fast as me telling a story as fast as I can make it up. I don’t see anything remotely bad about that. No one makes a career in the real world by copying someone else’s writing. There are tons of fan works but those do not make anyone real money and they only increase the reach of the original author.

    No, I think all the writers and artists hype was all about Altmann’s plan for a monopoly that got derailed when Yann LeCunn covertly leaked the Llama weights after Altmann went against the founding principles of OpenAI and made GPT3 proprietary.

    People got all upset about digital tools too back when they first came on the scene; about how they would destroy the artists. Sure it ended the era of hand painted cartoon cell animation, but it created stuff like Pixar.

    All of AI is a tool. The only thing to hate is this culture of reductionism where people are given free money in the form of great efficiency gains and they choose to do the same things with less people and cash out the free money instead of using the opportunity to offer more, expand, and do something new. A few people could get a great tool chain together and create a franchise greater, better planned, and more rich than anything corporations have ever done to date. The only thing to hate are these little regressive stupid people without vision, without motivation, and far too conservatively timid to take risks and create the future. We live in an age of cowards worthy of loathing. That is the only problem I see.

  • j4k3@lemmy.worldtoAsklemmy@lemmy.ml•Have you ever had someone get mad at you for asking why you shouldn't do something?English
    1·
    4 hours ago
    I use the term myth loosely in abstraction. Generalization of the tools of industry is still a mythos in an abstract sense. Someone with a new lathe they bought to bore the journals of an engine block has absolutely no connection or intentions related to class, workers, or society. That abstraction and assignment of meaning like a category or entity or class is simply the evolution of a divine mythos in the more complex humans of today.

    Stories about Skynet or The Matrix are about a similar struggle of the human class against machine gods. These have no relationship to the actual AI alignment problem and are instead a battle with more literal machine gods. Point is that the new thing is always the boogie man. Evolution must be deeply conservative most of the time. People display a similar trajectory of conservative aversion to change. In this light, the reasons for such resistance are largely irrelevant. It is a big change and will certainly get a lot of push back from conservative elements that collectively ensure change is not harmful. Those elements get cut off in the long term as the change propagates.

    You need a 16 GB or better GPU from the 30 series or higher, but then run Oobabooga text gen with the API and an 8×7b or like a 34b or 70b coder in a GGUF quantized model. Those are larger than most machines can run but Oobabooga can pull it off by splitting the model between CPU and GPU. You’ll just need the ram to initially load the thing or deepspeed to load it from NVME.

    Use a model with a long context and add a bunch of your chats into the prompt. Then ask for your user profile and start asking it questions about you that seem unrelated to any of your previous conversations in the context. You might be surprised by the results. Inference works both directions. You’re giving a lot of information that is specifically related to the ongoing interchanges and language choices. If you add a bunch of your social media posts, it is totally different in what the model will make up about you in a user profile. There is information of some sort that the model is capable of deciphering. It is not absolute or like some kind of conspiracy or trained behavior (I think), but the accuracy seemed uncanny to me. It spat out surprising information across multiple unrelated sessions when I tried it a year ago.

  • j4k3@lemmy.worldtoAsklemmy@lemmy.ml•Have you ever had someone get mad at you for asking why you shouldn't do something?English
    1·
    4 hours ago
    When tech changes quickly, some people always resist exponentially in the opposite vector. The bigger and more sudden the disruption, the bigger the push back.

    If you read some of Karl Marx stuff, it was the fear of the machines. Humans always make up a mythos of divine origin. Even atheists of the present are doing it. Almost all of the stories about AI are much the same stories of god machines that Marx was fearful of. There are many reasons why. Lemmy has several squeaky wheel users on this front. It is not a very good platform for sharing stuff about AI unfortunately.

    There are many reasons why AI is not a super effective solution and overused in many applications. Exploring uses and applications is the smart thing to be doing in the present. I play with it daily, but I will gatekeep over the use of any cloud based service. The information that can be gleaned from any interaction with an AI prompt is exponentially greater than any datamining stalkerware that existed prior. The real depth of this privacy evasive potential is only possible with a large number of individual interactions. So I expect all applications to interact with my self hosted OpenAI compatible server.

    The real frontier is in agentic workflows and developing effective niche focused momentum. Any addition of AI into general use type stuff is massively over used.

    Also people tend to make assumptions about code as if all devs are equal or capable. In some sense I am a dev, but not really. I’m more of a script kiddie that dabbles in assembly at times. I use AI more like stack exchange to good effect.

  • j4k3@lemmy.worldtoAsk Lemmy@lemmy.world•Has anybody here actually succeeded in getting an online conservative/right-winger to reflect and legitimately and long term shift their views left?English
    5·
    1 day ago

    You’ve still got time.

    I was in Geometry class when 9/11 happened. The day stopped. The news was turned on in class a few minutes before the second plane struck. I watched it in real time. I had been in those towers 6 months before too…

    About the worst rabbit holes for me were giving any audience to perpetual motion trolls, and Brown’s gas nonsense in car stuff.

    Everyone tries to simplify messy complexity and we are all tribal in scope. I’ve learned to only pay attention to people with academic credentials. I don’t watch translated nonsense from general news outlets. The information I pick up elsewhere is more collectivised where I expect to see a bunch of people talking about something from different angles before I view the information as relevant. I also do not care for any outlets claiming to bridge some divided narrative as these are controlling where the line in the sand is drawn. If two parties are Right and Right-Jihadists like in the USA, calling one party Left is manipulating by validating the status quo and outdated perspective.

    What changed me started with stratification of rock layers and realizing deep time was not compatible with my religious narrative. I encountered a sharp personal dislike for biases and prejudice against others without logic or reason. I encountered a lot of plausible seeming arguments, but ultimately the people making those arguments had nothing to offer; they are trolls with no depth, interests, personality, community, richness in life. Look at such a person’s profile and they are not real. There is no greater engagement or value they add to the world. All they do is make arguments that muddle political narratives. I learned to view these people as either getting paid to post or idiots. I care about real people and that means your politics should only ever be a small part of your person and profile. Any person that lacks a serious passion project and hobby(s) but posts their politics is a joke to me.

    In a way, I extend this to any group now. Like do people in your group include Nobel laureates that contribute significantly to the advancement of humanity. Because if they don’t, why bother wasting time with fools that lack top aspirations. Live life with no excuses. Excuses are for fools. Do the best you can with the cards you’re dealt in life.

  • j4k3@lemmy.worldtoAsk Lemmy@lemmy.world•How to avoid being a target of a psychopathEnglish
    6·
    3 days ago

    I know my weakness is my emotional depth. I keep protections in place mentally with a zero tolerance policy of trust. Like if my boss or coworker cheats on their partner, I know I will never trust that person or their character/ethics. The concept is one of the few that I retain from my religious past: “faithful in little; faithful in much.” Any person that shrugs off little lies or dishonesty is revealing their true ethics or lack there of. I do not try to hide who I am or lie about anything intentionally. Therein lies my lack of depth. I am unaware of how people filter and mask who they are in intentional ways, so anyone that shows harmful potential is someone I avoid and never trust more than is convenient or that I am forced.

  • j4k3@lemmy.worldtoPrivacy@lemmy.ml•Privacy Recommendations for a Young TeenEnglish
    11·
    4 days ago

    I think authoritarianism is a giant mistake and only creates duplicitous behavior. In my opinion tracking is ridiculous. None of us existed like this and ended up fine. In my opinion, all of this nonsense is acting as a stand in for relationships and real parenting. Humans make decisions and develop ethics based upon trust and autonomy. By stealing that factor of trust and autonomy, and replacing it with authoritarianism a parent is stunting the child’s growth of independent ethics and character. Make compelling discussions of why they should do whatever thing, but let them decide their own path. The lack of compelling discussions and real trust that requires risk is a major factor in the problems that exist in the present world.

    The one time you actually need to know where your kid is at because something has happened, you will not know because you have taught them that the only path to independence is to turn off the device and put it into a Faraday cage like pouch, or someone else will do so. If you have a fundamentally trusting relationship with open dialog and respect for their autonomy, they will tell you openly exactly where they are going and any potential for danger. If you can handle that information without allowing anxiety to overwhelm reasoning skills, you will be in a far better position to help them if something bad happens.

    The most long term valuable aspect of schooling is the development of one’s social network and connections, along with the habits and ethics. The actual information learned is rather limited in valuable application in the end. Who one knows and how one appears to others is of far more value than what one knows. For these reasons, there may be value in corporate social media. Simply teach the kid to understand how these places are both a trap and a tool. A trap, in that many of the smartest humans are manipulating users in ways that are nearly impossible for the users to escape. Never invest emotions into such a trap. Use the tool if needed for external social benefits, but use it as a manipulation tool with a layer of disconnect from who you really are. Teach them to use a work profile to isolate any apps from their device. That is just how I look at the issue.

  • j4k3@lemmy.worldtoADHD@lemmy.world•What do you think about r/ThisAintAdderall folks?English
    5·
    4 days ago

    My neighbor spent around 3 hours calling around to track down a source today. The wasted time and money that represents is the only crime I see. Don’t go talking about it or selling it, but whatever a person does in their kitchen for their own needs is none of my business. The wasted time is theft and therefore criminal actions against common people with no recourse. There are likely millions of victims of this crime.

  • j4k3@lemmy.worldtoOpen Source@lemmy.ml•Accessability of github for europeEnglish
    8·
    5 days ago

    Most key stuff is not on GitHub or GitHub is just a mirror. The heir apparent to Linux is Hartman and he moved to Europe a long time ago.

    No mobile devices are safe. Those are all proprietary black boxes for hardware. If the shit hits the fan, it is back to dumb phones and x86 computers. Digital doomsday prepers are not sounding all that crazy right now IMO.

    I have gotten weird interactions with rate limiting through GitHub because I will not whitelist their stalkerware collector server. They also pushed 2 factor to stalk and exploit through the only documented path they wanted people to take. I quit because of it.

  • j4k3@lemmy.worldOPtoBooks@lemmy.world•Microcontroller Exploits – Travis GoodspeedEnglish
    2·
    5 days ago

    Implant ROMs section is interesting. Even after all the work to get to the necessarily layer the difference between a 1 and 0 requires separating the R G and B wavelengths from the sensor and hoping the difference is pronounced enough in one of the channels that the bits can be mapped. With implant ROMs the bits are doped or not at the fab and there is no discernable difference. Once the ROM layer is revealed on the die, it requires Dash etching with 3 mL of 65% HNO3, 1 mL 48% HF, and 11 mL of 98% HAc.

    I’m not going to mess with no 48% hydrofluoric acid! Plus it takes a lower concentration HF to delayer the chip just to get to the implant ROM. After all of that, you can’t even see the difference with your own eyes. That is KGB level right there. The difference is only what is present in the central ‘O’ not the surrounding region in the image. The blue channel shows nothing, red is clearly defined, and green is less pronounced.

Moderates